Wait for lint/build before running E2E + Run snapshot check outside E2E - #8310
Merged
Merged
Conversation
…ight Two changes that stop non-E2E failures from surfacing as E2E failures: The E2E job now needs the type-check and bundle jobs, so a PR that does not compile never reaches Playwright. Previously a TypeScript break failed the E2E job's own build step and was counted as E2E flake. The `commands --tree` snapshot check moves from the Playwright suite to a plain node script (bin/check-commands-snapshot.js) that runs in the oclif-checks job next to the other regenerated-artifact checks. It needs no browser, no store, and no test org — it only runs the built CLI and diffs stdout against the committed snapshot. The snapshot file and the regenerate script stay where they were, so `pnpm test:regenerate-snapshots` works unchanged. Together these were ~12% of classified E2E-job failures over the last 90 days (196 snapshot + 112 build breaks), none of them E2E problems. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
github-actions
Bot
added
the
no-changelog
isaacroldan
changed the title
isaacroldan
changed the title
alfonso-noriega
approved these changes
Aug 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WHY are these changes introduced?
Analysis of 90 days of failed E2E shard logs showed ~12% of classified failures were not E2E problems at all: 196 were
commands --treesnapshot drift and 112 were TypeScript build breaks that failed the E2E job's own build step. Both surfaced as "E2E failure", polluting the signal of an already-noisy suite.WHAT is this pull request doing?
e2e-testsnowneeds: [type-check, bundle]. A PR that doesn't compile never reaches Playwright — the failure stays attributed to the build lane, and the E2E shards (plus their cleanup jobs) are skipped instead of failing.commands --treesnapshot check leaves the Playwright suite. It's nowbin/check-commands-snapshot.js— a plain node script that runs the built CLI and diffs stdout against the committed snapshot — executed in theoclif-checksjob next to the other regenerated-artifact checks (manifests, readme, docs). No browser, no store, no test org, and it fails fast with the same regeneration instructions as before plus a proper diff.The snapshot file (
packages/e2e/data/snapshots/commands.txt) andpnpm test:regenerate-snapshotsare unchanged.How to test your changes?
Measuring impact
🤖 Generated with Claude Code